Science Counts – But What Counts in Science?

We would like to discuss the collective interests of very different scientific disciplines. Therefore, we talked with leading researchers about their scientific aspirations and their confidence in scientific knowledge – we also asked them about obtaining, validating, and securing their results.

When is a result truly a result?

In science, reproducibility is one of the most crucial factors. “I try really hard not to get too excited when I see something for the first time,” says Graham Diering , PhD, Assistant Professor for biochemistry at the UNC Chapel Hill . “A result is a result only after I have reached it multiple times.”

Thomas Thannickal , PhD, Associate Research Professor at the Department of Psychiatry & Biobehavioral Sciences, University of California Los Angeles® , agrees: “The quality of the results depends on their reproducibility – i.e. the extent to which consistent results are obtained when an experiment is repeated.” And he makes a second point: “I don’t think a result can be fairly called a result without a very well designed experiment behind it. We always have our expectations about how the experiment will go. Sometimes we are wrong about our expectations, and we see something really surprising. This is always exciting, but only when we can have trust in the design of the experiment.”

For Talia N. Lerner , PhD, Assistant Professor of Physiology at Northwestern University® in Chicago, falsifiability is key: “Nothing is certain, ever. We work with hypotheses that are presumed true until proven false. If you haven’t formulated a falsifiable hypothesis, you aren’t doing science. For me, a ‘result’ is something that comes together when multiple lines of investigation point in the same direction. When very different experiments all fit into a model together, we begin to understand their meaning and can then form new hypotheses to test.”

In the laboratory, common analytical strategies – such as testing reproducibility, standardization and determining measurement accuracy - are part of the day-to-day business, says Prof. Dr. Boris Koch , specialist for ecological chemistry at the Alfred Wegener Institute Helmholtz Centre for Polar and Marine Research in Germany. “Though, for reliable evidence in marine chemistry and ecology and in questions with regard to climate change, long-time series with statistical verification of trends, process understanding, and simulation are further important elements.”

What do you do to optimize your quality of results?

As is the case everywhere in natural sciences, the documentation of methods is of the utmost importance in order to ensure reproducibility by other researchers, stresses Boris Koch. For him and his colleagues who work on the high seas, there are still some very special aspects to consider: “Sampling for a trace analysis on board of a research vessel poses a particular challenge. One example is the analysis of dissolved iron in seawater, with concentrations in the lower nanomol range and many sources of contamination at the same time. In addition, the chemical work on a ship is especially complicated – for instance, it is not possible to weigh anything on a ship. We therefore have to weigh chemicals in advance of the expedition or, if possible, pipette them on board.” And because many measuring instruments are sensitive to vibrations, Koch and his team have to carry out a lot of their analyses in their laboratory ashore. This makes sample conservation a crucial point.

“All researchers build on each other’s work,” emphasizes Talia Lerner. “If we get something wrong, others will spend time going down a wrong path, too, or trying to prove us wrong. Ultimately, the quality of our results determines how impactful they will be.” This is why quality in the laboratory matters all the time, at every single step. “If we want to help human patients, we need to be sure we are getting things right. We need to imagine how our results would translate into practical therapies. Proper controlling and careful notetaking are key”, says Lerner. “I rely on my lab members to be careful observers, always looking for reasons an experiment might be contaminated.” Her team does a lot to avoid possible systemic and measurement errors: “We can help each other do experiments in a blinded fashion, set concrete inclusion/exclusion criteria prior to analysis, and scrutinize each other’s work for flaws. And we work hard to verify that all our reagents are of high quality and our handling and analysis methods are standardized .” Though, of course, it’s always possible that there is something quirky about the lab, says Lerner. “In the end, cross-validation of our results by other groups is important.”

For Graham Diering, the validation process begins with replication: “First of all, we repeat the experiment. Quality results have to be replicable. ‘Guaranteeing’ the quality of results is not always easy, but we try to isolate as many different variables as we possibly can in our experimental design. That makes the experiment easier to interpret and easier to replicate.” Keeping that in mind and trying to reduce all possible sources of error at the lab, there is one single factor you should focus on: “The biggest variable in scientific experiments is the experimenter. A very careful design of experiments and an equally careful execution of those experiments are critical for quality results. The best way to strengthen any claim in science is to have the results replicated by different labs, ideally even in different parts of the world.”

As different laboratories are equipped in different ways, it is often less important or helpful to use the newest or fanciest methods or equipment in trying to replicate each other’s results, says Diering: “I really like to use reliable and time-tested ‘oldfashioned’ methods whenever I can.”

With regard to environmental and climate research, the scope of available data and samples is of particular importance, Boris Koch explains: “ Automated measurement methods are particularly helpful because they can provide a much more comprehensive picture of changes over time, for example.”

“We need reliable laboratory equipment and – as I have mentioned – that allows for automation of manual labor”, says Talia Lerner. She stresses that “everyday lab work can be repetitive and boring, but careful science depends on standardization . Some help comes from automation – increasingly, we can make computer programs and robots take over our manual labor and give us more time for creative thought. Until then, podcasts and labmates you enjoy hanging out with are lifesavers.”

On the other hand, the rapid technological progress means a growing funding problem, as Thomas Thannickal points out: “For us, a main challenge is the rapid change in scientific methods and instruments. There is very rapid change in neuroscience right now. We are not able to adapt new instrumentations with limited funding.” But this can be overcome by using shared facilities and working collaboratively, Thannickal suggests.

What makes science trustworthy?

Some people don’t trust scientists, thinking they would fake experiments and work with made-up results. This has little to do with reality, though, if you ask people working in research. “Science is trustworthy because we test our hypotheses,” explains Talia Lerner. “We are open to many explanations of the data we collect – and we actively try to determine which explanation might fit best our observations. While we are never 100 percent certain of a result – there could always be something we missed or an alternative explanation we have not explored – we are always willing to examine things critically.”

The field of climate research is extremely complex and a particular challenge is the distinction between natural and man-made changes, admits Koch. But in the past decades, an enormous number of studies and long-term studies have been carried out that allow a broad scientific consensus on the influence of humans on our climate. They deal with independent systems and are supported by a look at climate archives such as ice and sediments and by Earth system models, says Koch. “It is important to interpret the significance of results correctly and in the right context. For instance, a common question would be: to what extent is a finding that I have made at a particular time in a particular marine region representative of the global ocean?” According to Koch, the degree to which results are representative is in turn related to the volume and comparability of the available data. “For this reason, marine researchers today also have a great responsibility to make the data available and provide open access to them.”

Science only progresses when scientists check each other’s work, if they replicate somebody else’s experiment and build on the foundations set by their predecessors. Lerner sums it up: “People should never trust one study (scientists don’t!), but they can trust that the enterprise of science as a whole builds towards truth.” Graham Diering feels the same way: “Opinions matter very little with regard to scientific results. Science is all about asking questions, making observations and testing hypotheses.” He stresses, “I always have my opinions on how an experiment will turn out. This is based on many years of training, but in the end, I am just a biologist. I don’t get to tell biology how it works; biology tells me how it works.”

For more information about the scientists click HERE